2024-08-26 23:13:49,032 [INFO] (BaseMetadataBackend) Enforcing minimum image size of 0.0. This could take a while for very-large datasets.
                                                                                                                                                                                                                                                                                                  2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 0.68 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=4).                                                                 | 0/23 [00:00<?, ?it/s]
2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 1.46 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=4).
2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 0.57 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 1.54 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=3).
2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 2.18 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=3).
2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 2.5 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,033 [WARNING] (BaseMetadataBackend) Removing bucket 1.36 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=6).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 2.09 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 0.74 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=3).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 0.94 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 0.65 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=2).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 0.5 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 1.13 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=2).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 2.6 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 0.6 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 1.83 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,034 [WARNING] (BaseMetadataBackend) Removing bucket 1.2 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,035 [WARNING] (BaseMetadataBackend) Removing bucket 0.48 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
2024-08-26 23:13:49,035 [WARNING] (BaseMetadataBackend) Removing bucket 1.75 due to insufficient samples; your batch size may be too large for the small quantity of data (batch_size=8 > sample_count=1).
                                                                                                                                                                                                                                                                                                  2024-08-26 23:13:49,035 [INFO] (BaseMetadataBackend) Completed aspect bucket update.                                                                                                                                                                                                              
2024-08-26 23:13:49,036 [INFO] (__main__) Reloading cache for backend something-special-to-remember-by
2024-08-26 23:13:49,036 [INFO] (DiscoveryMetadataBackend) Checking for cache file: /root/simpletuner/data/aspect_ratio_bucket_indices_something-special-to-remember-by.json
2024-08-26 23:13:49,036 [INFO] (DiscoveryMetadataBackend) Pulling cache file from storage
2024-08-26 23:13:49,036 [INFO] (__main__) Waiting for other threads to finish..
2024-08-26 23:13:49,036 [INFO] (__main__) Splitting buckets across GPUs
2024-08-26 23:13:49,037 [INFO] (__main__) Rebuilding VAE cache..
(id=something-special-to-remember-by) Bucket 1.29 caching results: {'not_local': 0, 'already_cached': 0, 'cached': 0, 'total': 17}
(id=something-special-to-remember-by) Bucket 1.0 caching results: {'not_local': 0, 'already_cached': 0, 'cached': 0, 'total': 34}
(id=something-special-to-remember-by) Bucket 0.88 caching results: {'not_local': 0, 'already_cached': 0, 'cached': 0, 'total': 8}
(id=something-special-to-remember-by) Bucket 0.78 caching results: {'not_local': 0, 'already_cached': 0, 'cached': 1, 'total': 65}
Epoch 4/556, Steps:   1%|▏                                | 64/10000 [07:35<15:10:45,  5.50s/it, lr=6.4e-7, step_loss=0.0806]